Skip to content

feat(testing): add skip-time virtual clock#523

Merged
yaythomas merged 1 commit into
mainfrom
feat/skip-time-clock
Jul 14, 2026
Merged

feat(testing): add skip-time virtual clock#523
yaythomas merged 1 commit into
mainfrom
feat/skip-time-clock

Conversation

@yaythomas

@yaythomas yaythomas commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Description of changes:

  • Add Clock protocol with RealClock and SkipClock implementations
  • Give each ExecutionWorker its own clock via the registry factory
  • Drive wait and step-retry timers off the clock so modeled delays complete instantly under skip while history keeps real durations
  • Default DurableFunctionTestRunner to skip_time=True; pass skip_time=False for real wall-clock timing
  • Remove the DURABLE_EXECUTION_TIME_SCALE env-var scaling

Issue #, if available:
closes #437

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@yaythomas yaythomas mentioned this pull request Jul 9, 2026
Closed
@yaythomas yaythomas force-pushed the feat/per-execution-worker branch from 5c208f2 to 355edf1 Compare July 9, 2026 08:49
@yaythomas yaythomas force-pushed the feat/skip-time-clock branch from 346a870 to 2e9ba2b Compare July 9, 2026 08:56
@yaythomas yaythomas force-pushed the feat/per-execution-worker branch 2 times, most recently from 9829546 to 71962f1 Compare July 9, 2026 14:50
@yaythomas yaythomas force-pushed the feat/skip-time-clock branch 2 times, most recently from d2f04f2 to c494657 Compare July 9, 2026 15:27
@yaythomas yaythomas force-pushed the feat/per-execution-worker branch from 71962f1 to 11103e3 Compare July 10, 2026 00:32
@yaythomas yaythomas force-pushed the feat/skip-time-clock branch from c494657 to c1b3e66 Compare July 10, 2026 09:18
zhongkechen
zhongkechen previously approved these changes Jul 10, 2026
View reviewed changes

@zhongkechen zhongkechen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yaythomas yaythomas mentioned this pull request Jul 13, 2026
Open
Base automatically changed from feat/per-execution-worker to main July 13, 2026 20:15
@yaythomas yaythomas dismissed zhongkechen’s stale review July 13, 2026 20:15

The base branch was changed.

@yaythomas yaythomas force-pushed the feat/skip-time-clock branch from c1b3e66 to c317391 Compare July 13, 2026 20:19
@yaythomas yaythomas requested a review from zhongkechen July 13, 2026 20:20
@yaythomas

yaythomas commented Jul 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

rebased on main, no other change

[edit:] ooh hang on CI failures 😕 , investigating

zhongkechen
zhongkechen previously approved these changes Jul 13, 2026
View reviewed changes
@yaythomas yaythomas dismissed zhongkechen’s stale review July 13, 2026 20:37

failing tests, fix inbound

@zhongkechen

zhongkechen commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

seems wait_for_callback(...) failed before the test ever sends a callback.

@yaythomas yaythomas force-pushed the feat/skip-time-clock branch 2 times, most recently from 907f424 to 089c4b8 Compare July 14, 2026 00:30
@yaythomas

yaythomas commented Jul 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

changed:

  1. Advancing SkipClock: now() = real time + accumulated skip, arm() adds the gap — monotonic, distinct timestamps, matching fake-timers shouldAdvanceTime: true.
  2. Per-runner clock: registry and worker are clock-free again; one Clock is shared by the Executor, CheckpointProcessor, and both runner types.
  3. All history stamp sites moved to the clock: execution start/end, wait and callback completions, invocation start/end, the history fallback, and the stop response now reuses the stamped time. events_to_operations uses the event's own timestamp instead of wall-clock. Test-only fallbacks go through real_now().
  4. TID251 ruff ban on datetime.now outside clock.py — verified it fires on a synthetic violation.

Verification: 1353 testing-package tests pass, full workspace run 2883 passed, the 4 previously failing callback tests pass locally and in CI, mypy clean (119 files), hatch fmt --check clean.

3.13 integration test failure failing only on test_otel_logger_example_spans_in_xray, which is known pre-existing OTel X-Ray flk.

@yaythomas yaythomas requested a review from zhongkechen July 14, 2026 01:28
@yaythomas yaythomas force-pushed the feat/skip-time-clock branch from 089c4b8 to da05414 Compare July 14, 2026 18:41
zhongkechen
zhongkechen previously approved these changes Jul 14, 2026
View reviewed changes
Comment thread ...-durable-execution-sdk-python-testing/src/aws_durable_execution_sdk_python_testing/runner.py Outdated
Comment thread ...-durable-execution-sdk-python-testing/src/aws_durable_execution_sdk_python_testing/runner.py
store if store is not None else InMemoryExecutionStore()
)
self.poll_interval = poll_interval
self._clock: Clock = SkipClock() if skip_time else RealClock()

@zhongkechen zhongkechen Jul 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would a shared SkipClock() between multiple executions cause issues?

@yaythomas yaythomas Jul 14, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only cross-execution effect is acceleration.

If execution A arms a skip past execution B's wait horizon, B's wait becomes due on its next wake instead of at its own arm. Since a single monotonic clock serves everything, it can only accelerate, never reorder.

What it can't do:

  1. Prematurely fire externally driven timers. Callback timeout/heartbeat, execution timeout, and invocation timeout are real call_later delays on the scheduler, so A skipping an hour of virtual time doesn't age B's 30s callback timeout.
  2. Distort history. Modeled durations come from scheduled_end - start, both stamped from the same clock, so they stay faithful regardless of who skipped what.
  3. The alternative (per-execution clocks) is worse: executions share one store and one history surface, so independently advancing clocks put cross-execution timestamps in different clock domains — the same class of bug as the CI failure this revision fixed, one level up.

@yaythomas
feat(testing): add skip-time virtual clock
983e4bb 
- Add Clock protocol with RealClock and SkipClock implementations
- SkipClock returns real time plus the accumulated skipped duration,
  so virtual time advances with real time between jumps and every
  read stays monotonic and distinct
- One clock per runner, shared by the Executor, CheckpointProcessor,
  and all timestamp stamping, so history events form a single totally
  ordered clock domain
- Drive wait and step-retry timers off the clock so modeled delays
  complete instantly under skip while history keeps modeled durations
- Stamp operation, invocation, and execution start/end timestamps
  from the clock; reconstruct operations from event timestamps
- Ban datetime.now via ruff TID251 outside clock.py so all time reads
  go through the clock
- Default DurableFunctionTestRunner to skip_time=True; pass
  skip_time=False for real wall-clock timing
- Remove the DURABLE_EXECUTION_TIME_SCALE env-var scaling
@yaythomas yaythomas force-pushed the feat/skip-time-clock branch from da05414 to 983e4bb Compare July 14, 2026 20:19
@yaythomas yaythomas requested a review from zhongkechen July 14, 2026 20:39
@yaythomas yaythomas merged commit 887eb4a into main Jul 14, 2026
12 checks passed
@yaythomas yaythomas deleted the feat/skip-time-clock branch July 14, 2026 20:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zhongkechen zhongkechen zhongkechen approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Testing] [Feature]: Add option to skip time in test runner

2 participants

@yaythomas @zhongkechen